from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.017	0.041	0.000	0.002	-1	1	0.676	20.326	0.126	0.676	0.099	0.099	See	See
1	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.003	0.000	0.023	-1	1	0.000	0.354	0.010	0.000	0.066	0.066	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.106	0.046	0.000	0.003	-1	5	0.743	20.393	0.027	0.743	0.152	0.152	See	See
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.002	0.000	0.025	-1	5	1.000	0.354	0.011	1.000	0.070	0.070	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.384	0.005	0.000	0.002	1	100	0.846	20.109	0.020	0.846	0.119	0.119	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	0.000	0.021	1	100	1.000	0.367	0.008	1.000	0.056	0.056	See	See
6	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.119	0.050	0.000	0.003	-1	100	0.846	20.266	0.195	0.846	0.154	0.154	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.004	0.000	0.026	-1	100	1.000	0.363	0.007	1.000	0.071	0.071	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.375	0.005	0.000	0.002	1	5	0.743	20.052	0.041	0.743	0.118	0.118	See	See
9	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.000	0.000	0.020	1	5	1.000	0.365	0.010	1.000	0.055	0.055	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.252	0.016	0.001	0.001	1	1	0.676	20.651	0.200	0.676	0.061	0.061	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.019	0.000	0.000	0.019	1	1	0.000	0.351	0.010	0.000	0.055	0.055	See	See
12	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.744	0.032	0.000	0.002	-1	1	0.845	4.357	0.011	0.845	0.400	0.400	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1	2	0.008	0.002	0.000	0.008	-1	1	1.000	0.286	0.005	1.000	0.028	0.028	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.964	0.022	0.000	0.003	-1	5	0.883	4.287	0.022	0.883	0.691	0.691	See	See
15	KNeighborsClassifier_brute_force	predict	100000	1	2	0.011	0.005	0.000	0.011	-1	5	1.000	0.289	0.005	1.000	0.037	0.037	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.311	0.002	0.000	0.002	1	100	0.887	4.328	0.009	0.887	0.534	0.534	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.000	0.000	0.004	1	100	1.000	0.286	0.004	1.000	0.012	0.012	See	See
18	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.991	0.038	0.000	0.003	-1	100	0.887	4.341	0.026	0.887	0.689	0.689	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1	2	0.010	0.003	0.000	0.010	-1	100	1.000	0.287	0.005	1.000	0.036	0.036	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.337	0.027	0.000	0.002	1	5	0.883	4.281	0.024	0.883	0.546	0.546	See	See
21	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.000	0.000	0.004	1	5	1.000	0.287	0.006	1.000	0.012	0.012	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.113	0.007	0.000	0.001	1	1	0.845	4.347	0.012	0.845	0.256	0.256	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.002	0.000	0.000	0.002	1	1	1.000	0.287	0.004	1.000	0.008	0.008	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.699	0.009	0.000	0.002	1	5	0.975	122.443	0.000	0.975	0.014	0.014	See	See
1	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	1.000	3.060	0.028	1.000	0.001	0.001	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.967	0.011	0.000	0.001	-1	5	0.975	121.677	0.000	0.975	0.008	0.008	See	See
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	1.000	3.067	0.049	1.000	0.001	0.001	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.512	0.007	0.000	0.001	-1	1	0.964	122.636	0.000	0.964	0.004	0.004	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	0.000	0.003	-1	1	1.000	3.075	0.042	1.000	0.001	0.001	See	See
6	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.892	0.015	0.000	0.001	1	1	0.964	127.365	0.000	0.964	0.007	0.007	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	1.000	3.071	0.069	1.000	0.000	0.000	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.390	0.058	0.000	0.005	1	100	0.973	126.638	0.000	0.973	0.043	0.043	See	See
9	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.000	0.004	1	100	1.000	3.103	0.082	1.000	0.001	0.001	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	2.928	0.024	0.000	0.003	-1	100	0.973	127.925	0.000	0.973	0.023	0.023	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	1.000	3.073	0.071	1.000	0.002	0.002	See	See
12	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.023	0.000	0.001	0.000	1	5	0.923	0.047	0.001	0.923	0.493	0.493	See	See
13	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	5	1.000	0.006	0.000	1.000	0.110	0.110	See	See
14	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.026	0.000	0.001	0.000	-1	5	0.923	0.046	0.000	0.923	0.564	0.564	See	See
15	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	1.000	0.006	0.000	1.000	0.386	0.386	See	See
16	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.000	0.001	0.000	-1	1	0.895	0.045	0.000	0.895	0.535	0.535	See	See
17	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.001	0.000	0.003	-1	1	1.000	0.006	0.000	1.000	0.421	0.421	See	See
18	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.021	0.000	0.001	0.000	1	1	0.895	0.045	0.000	0.895	0.473	0.473	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	1	1.000	0.006	0.000	1.000	0.112	0.112	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.039	0.000	0.000	0.000	1	100	0.919	0.073	0.003	0.919	0.536	0.536	See	See
21	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	100	1.000	0.006	0.000	1.000	0.113	0.113	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.040	0.000	0.000	0.000	-1	100	0.919	0.071	0.000	0.919	0.563	0.563	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	100	1.000	0.006	0.000	1.000	0.394	0.394	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.144	0.002	300	0.006	0.000	0.798	0.520	0.020	0.798	0.277	0.277	See	See
1	HistGradientBoostingClassifier_best	predict	100000	1	100	0.019	0.001	300	0.000	0.019	1.000	0.432	0.013	1.000	0.043	0.043	See	See